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Summary :  Using  the  teohnlque  of  continuous 
approximation,  approximate  solutions  to  a 
number  of  important  multi-stage  scheduling 
problems  are  determined.  In  addition,  the 
functional  equation  approach  of  the  theory 
of  dynamic  programming  Is  used  to  derive  an 
alternate  proof. of  a  result  of  S.  Johnson, 
contained  In  P—402. 


A 


Some  Comblna torlal  Problems  Arising  in  the  Theory 
of  Multi-btage  Processes 

by 

Richard  bellman  and  uliver  uro  s 

1 u  lntroductl on, 

A  problem  of  some  import  ince  in  industrial  applications  which 
gives  rise  to  some  interesting  and  quite  difficult  combinot orial  pro¬ 
blems  is  the  following: 

"There  ore  n  items,  not  all  identical,  which  have  t o  be  processed 
through  a  number  of  machines  of  different  type.  The  order  In  which  the 
machines  are  to  be  used  la  not  immaterial,  since  some  processes  must  be 
carried  out  before  others.  Given  the  times  required  by  the  i-th  item  on 
the  J-th  machine,  «ij,  i  «  l,2,.««,n,  j*l,2,..„,m,  determine  the  order 
in  which  the  items  should  be  fed  into  the  machines  sd  as  to  minimize  the 
total  time  required  to  complete  the  lot„" 

As  a  simple  example  of  the  above,  consider  the  case  where  we  have 
a  number  of  books  which  must  be  printed  and  then  bo  undo  Clearly  the 
printing  must  precede  the  binding.. 

Mathematically,  the  j robiem  is  one  of  arrangements,  which  can  be 
solved  in  any  ] articular  case  by  enumeration,  however,  a  juicK  count 
of  the  josaible  a rraiigem ent s  will  show  that  as  soon  ns  the  number  of 
items  reaches  ten,  the  e  numerat  ive  techni  ue  becomes  unwieldy,, 

At  tne  suggestion  of  one  of  the  authors,  this  problem  was  investi¬ 
gated  oy  .J.  Johnson,  who  found  a  number  of  interesting  and  important 
results  which  we  sic  11  desciit6  below,  .he  ..as  able  to  solve  coaipletely 
the  two- st.  ge  problem  for  any  number  of  dissimilar  items.  The  three 
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stage  problem,  however,  poses  difficulties  of  an  entirely  new  and 
formidable  type,  and  ties  so  fnr  resisted  solution  „ 

An  important  tool  in  Johnson’s  work  waa  an  explicit  formula  for 
idle  time  on  the  second  machine  in  the  two  stage  process  „  It  is  this 
quantity  which  determines  the  efficiency  of  a  particular  routing  order „ 

In  this  p«per  we  shall  tackle  the  simplified  problem  of  determining 
the  optimal  order  when  there  are  a  large  number  of  items  of  only  a  few 
different  types,,  This  is  perhaps  the  most  important  problem  as  far  as 
practice  is  concerned,, 

Even  here,  the  original  problem  .seems  difficult  to  resolve.  Con¬ 
sequently  we  shall  use  a  device  which  works  uniformly  well  throughout 
the  theory  of  dynamic  programming,  namely  the  replacement  of  a  discrete 
problem  by  a  continuous  version*  As  is  fre  ufcntly  the  cast,  the  continu¬ 
ous  version  may  be  solved  with  great  ease  and  elegance* 

For  those  Interested  in  other  problems  involving  multi-stage  pro¬ 
cesses  which  may  be  approximated  by  continuous  proces  63  and  thereby 
resolved,  we  refer  to  Cl,  C*  J  nnd  <-5j  f  whrre  otner  references  to 
the  theory  of  uy  mimic  pr oprivui  lng  muj  be  found, 

Finally  we  ahall  show  how  the  two-rtage  process  may  be  attacked 
by  the  functional  etuation  approach  of  the  theory  of  dynamic  programming, 
cf  CO  ,  and  resolved  without  the  use  of  an  explicit  1'oimuia  for 

tne  idle  time  Tills  method  is  important  since  it  is  not  always  possible 
to  obtain  a  tractable  explicit  analytic  representation  of  the  quantity 
that  is  to  be  minimized  or  maximized  In  many  analogous  problems* 
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The  methods  presented  here  may  also  be  used  to  to  treat  the 
case  where  there  are  1  nt6rch  an,  eab  le  machines,  operators  trained  to 
work  soiiie  or  all  of  the  machines,  and  so  on.  We  feel  that  It  is  best 
to  expose  the  method  in  its  native  simplicity  and  leave  the  extension:* 
to  those  to  whom  the  j  roblem  may  have  some  immediate  interest. 

1  he  hesuit  s  of  J  ohn non . 

In  tliin  section  we  shall  present  the  results  already  obtained  by 
oohnson.  he  first  of  all  established  b.r  a  simple  argument  that  for  the 
cases  of  two  nnd  tiiree  machines  the  order  of  processing  on  each  machine 
can  be  tnxen  to  be  the  same  in  un  optimal  arra  ngem  ent ,  This  resuits  in 
a  considerable  simplification  and  permits  an  explicit  formula  for  the 
idle  time  to  be  de  rived  *  M  % 

Let  us  note  in  passing  that  .Johnson  showed  by  means  of  a  simple  ex¬ 
ample  that  this  identical  ordering  may  not  be  valid  for  four  or  more 
s  tu  gee  „ 

Theorem  ll Johnson  )„  Let  xL  be  the  inactive  t  lme  on  the  second  machln  e 
immediately  before  the  i-th  1  tern  is  pr  ocesaed  on  the  second  machine  „ 

Let  (a^.b^j  be  the  t lmea  re  4ulred  to  \  rocess  t  he  i-th  item  on  the  first 
and  second  much lne  s  respectively,  and  as  sume  that  the  it  ems  are  arranged 
in  numerical  order.  Then 


w  hare  lz<v  represents  tne  total  idle  time  on  t  ha  sec  ond  ma  ch  i  ne  „ 

.ve  shall  give  the  proof  for  the  saKe  of  completeness,  since  it  is 


quite  snort. 
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The  remainder  of  th«  proof  la  Induct  ive0 

The  two  machine  problem  la  equivalent  to  determining  the  arrange¬ 
ment  of  n  ltema  which  will  minimize  the  rlf.ht  side  of  (l)o  The  solution 
la  filven  by 

Theorem  2(  J  onn.jon  i An  optimal  ordering  1  a  determined  b^  th e  following 
rule :  lt6m  1  p recedes  item  j  If 

(t)  min  la  -  ,b-  ;  <  min  [a-  ,b ■  j „ 

J  vt 

If  the  re  Is  equal  l  ty  ,  el  t  ne  r  order  l  n^  ia  optimal  t  i  rov  lded.  that  1 1 
1  s  c  onsl  ste  nt  with  all  the  definite  p  r»  f  e  re  nc  a  a 
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Kor  the  three-stape  problem,  the  c  or  re  3  pond  \  ng  formula  for  the 
total  idle  time  on  the  third  machine,  also  derived  by  Johnson,  is 


(7)  .1, 


w  tie  re  ,b;,C; 

on  the  first* 


|{U  i  V  i  tv 


V- 


2 


:>*.  ♦  z  k.  - 


,  denote  respectively  the  times  required  by 
second  a  nd  third  niachineso 


the  ith  item 


3o  Continuous  Version s  -  2  machines,, 


To  illustrate  our  technics,  let  ua  begin  by  considering  the 
simplest  case,  that  of  two  machines  and  two  different  types  of  ltemsj 
assuming  that  the  total  number  of  items  is  large  compared  to  the  times 
required  to  process  any  individual  item.  In  place  of  the  expression 

a  »*• 

2«.  -  lb-  ,  we  consider  the  integral 
(1)  I  (  u)  =  ^(e(t)  -  bit)  )  d  t . 


The  analogue  of  an  ui rangement  of  items  is  a  characteristic  function 
i>  defined  over  th«  interval  Qj ,  tJ  .  This  function  determines  ait)  and 
bit)  in  the  following  way 


12) 


alt)  = 

o ,  4>  ♦  u ,  0  -4>) 

b(t)  - 

M  ♦ 

where  (u,  , b,  )  and  ial(bx)  represent  the  tiueo  required  on  the  first  and 
second  machines  for  t he  first  anc  second  types  respectively.  The  function 
i  (t)  is  the  characteristic  function  of  the  oet  over  whicn  the  first  item 
is  jrocessed,  and  i- is  the  characteristic  function  of  the  eet  over 
the  second  item  is  j recessed  The  constraints  upon  <p  are  that  it  take 
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on  only  the  values  C  and  1,  and  in  addition  satisfy 

T 

13)  \  t-Ur-St, 

0 

which  is  equivalent  to  the  statement  that  k  of  the  T  Items  belong  to 
the  first  type.  If  we  set 

«  =  (.  —  * ».  +  ia*  fc  t  ) 


14) 


?  * 


the  problem  Is  that  of  determining  the  quantity 

(5)  I  =  Min  Max  F  *  ^  4**  ♦  ?! 

4  of  T  U  J  3 

and  determining  the  corresponding  functions 

It  Is  to  be  expect  ad  that  in  general  to  obtain  a  minimum  we  must  ex¬ 
tend  our  class  of  functions  and  consider  those  which  correspond  to  "mixed  ' 
policies.  This  is  to  say  we  must  allow  4  to  satisfy  the  weaker  condition 
0  «  $  .*■  1,  ratl«r  than  be  restricted  to  values  of  0  or  1.  For  two  machines, 
there  is  a  solution  in  the  narrower  class.  For  three  machines,  this  will 
not  be  true  in  the  majority  of  cases.  This  illustrates  the  complexity  of 
the  solution  of  t  he  corresponding  discrete  problems  , 

Let  U3  now  demonstrate 
Th6  or  am  3«>  A  minimizing  $  l_s  given  by 


( 6 )  If  o<  >  o  } 

$  *  ( 

in 

LT-^'rJ 

• 

4  -  o 

in 

l°>  T-*J 

If  «.<  oJ 

4  -  l 

in 

r-i 

* 

o 

-  o 

in 

If  oigj  p  is  arbitra ry . 
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Proof :  Let  us  consider  *>G  first.  Denote  the  <f>  described  above 
by  «  Then  for  any  4  satisfying  (3)  and  Os  £*1,  it  is  clear  that 

a  -  ^  p 

(  7 )  c*  's  4*  ¥  ^  x  i  <*  Su  4  JX  4  ^U. 


Hence 

( B) 

O  «  u  ST 

which  shows  that  4  furnishes  the  minimum, 
treated  similarly.,  It  is  clear  that  for  **0, 
completes  the  proof. 


4  At  ♦ 


a.Mjr 


L 


The  case  where  <  0  is 
4  plays  no  role.  This 


The  4  we  have  found  is  not  necessarily  unique.  It  is,  however,  the 
simplest  solution,  and  the  most  important  in  applications. 


Let  us  now  consider  the  error  made  in  U3ing  the  Solution  to  the 

continuous  case  as  an  approximation  the  solution  of  the  discrete  oase. 

*.  •!»  *  * 

We  S6Q  that  the  difference  between  ^  ^  and  "  . 

if,  *  *  *  V  •  I  » r  I 

is  bv  or  b^  ,  hence  ne  eligible  if  n  is  large.  Now  consider  the  difference 
between 


(9)  8 


a  ■*.  -id 

*Z.  ^  -  *2.  b:  *  ^  [( v  b»)i;  *  ^ 

»  1  .  ,  !  »  1  •  is  1 


u»  1,2 . .  where  4>^  is  1  if  the  first  item  occupies  the  i-th  position 

and  zero  otherwise,  and  the  continuous  analogue  j  IT(u)  : 

V  ' 

It  is  clearly  sufficient  to  consider  the  difference  at  integral  values 
of  Uo  We  have 

( ic )  J (  a)  -  X  U)  -  x  (  ^  4  ^  4,  )  . 

For  each  4,  consider  the  corresponding  41*3  which  has  the  property 

*.  A 

that  S  ^  4  ,  ,  whe  n  u  =  1 ,2, . .  c. ,  n.  The  d  if  fere  nee  will  then  be  zero. 

«•  ’  >1 
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It  follows  than  that  if  we  minimize  J(u)  over  all  4  and  find  that 

the  minimizing  f  ,  or  a  minimizing  ,  corresponds  to  a  ^  ,  wa  obtain 
a  vary  good  approximation  to  the  solution  of  the  original  problem,,  Even, 
as  below,  in  the  case  wh6re  there  is  no  minimizing  function  corresponding 
to  a  characteristic  function,  we  may  still  be  able  to  approximate  by  means 
of  characteristic  functions,  and  obtain  a  useful  answer,,  For  example,  if 
t>  -  a,  whe  re  0<  a  <  1,  in  the  interval  (G,s) ,  we  may  consider  an  approximate 
'V  given  by  t-1  in  (0,a^  and  G  in  (a,s/.  Still  finer  subdivisions  will 
yield  more  accurate  approximation.  For  a  discussion  of  this  concept  of 
replacing  mixed  strategies  by  pure  strategies,  see  ;  for  an  application 
see  Ztt  o 


The  case  where  there  are  more  than  two  types  of  items  is  treated 
in  a  similar  manner.  Taive,  for  instance,  the  case  of  three  types  of 
items.  Let  be  the  respective  characteristic  functions  of  the 

sets  where  eacn  is  processed  .  Then,  corresponding  to  1 5)  we  have  the 
problem  of  determining  the  minimum  of 

It. 

(11  i  I(4>)*  f  ^  (vfei )4>iJ  IX 

«  T  L  O 


over  ail 

(12) 


4-,$^  and  satisfying  the  constraints, 

o  s  W  I 

(fc)  <*!♦  4-,  *  I  T  -r 

C->  ^  ♦,  AX  S  Jk(>  s  ,  S  Lt  =  -fcj  j 

®  0 

jtf,  ♦  =  T 


Using  the  result  of  (7),  it  is  clear  that  the  order  of  the  items 
is  determined  by  the  quantities  a;-  b;«d;  ,  which  are  to  be  arranged 
in  increasing  order  of  magintude.  This  yields  th6  optimal  ordering. 
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4.  Continuous  Version.  Three  Machines 0 

Let  us  now  discuss  the  three-machine  problem  assuming  two  distinct 
varieties  of  items.  Th6  method  given  here  extends  to  any  number  of  stage 
and  any  -number  of  items,  provided  that  we  assume  that  the  ordering  on 
each  machine  is  the  smam  . 

As  the  continuous  analogue  of  th6  expression  given  for  1^  in  12.7) 
we  have 

(1)  I.  =  max  r  *  -> 

VT;  o«  um<  T  L  y  a  1 1 1  -bi  t )  )dt  +  ^ lblt)-clt) )dt^ 

o  o 

<  * 

•  max  f  ^  t  (4  u.  * 

oSMSviT*"  *  “ 


upon  setting 

12)  aCt.^  o, 4>  ■»  <»v(‘-4>>>  W(t)-  b'4>  +  bv(i- tit)-  c,4t  tvl'-*)j 

and 

(3)  ♦  ba-  b, ,  ?  *  t  *  b'“  b*-»  ^-c*,  *  *  4»% 


Me  wi  sh  to  determine  the  minimum  of  I  for  all  ^  subject  to 

*  (T) 


A  solution  is  given  by 


Th6 or em  4.  The  minimum  value  of  1  is 

(5)  vivo-  ■*«*  ( 

A  minimizing  i_s  gl van  by 

( 6  j  */"r 

for  C  4  t  i  T.  In  gen  eral.  the  solution  is  non- unique. 
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rroof:  We  have 


17)  i  o  . 

—  f  Ok  .  i  T 

> 

^  ♦  U^)T 

j 

Or.  the  other  hand,  if  wi  take  4  =  k/r, 
of  the  linear  form 


setting  u  v  C. 
setting  a  n,v  T„ 
setting  u  T,v  T. 
we  wi3h  to  determine  the  maximum 


(8) 


L  tvgx) 


♦  * 


u  + 


■«  Jfcv 


over  the  region  G<  u*  v  i  T.  The  maximum  occurs  at  the  of  the  vertices 
1 0 , 0  J  ,  10, T),  or  (T,Tj,  and  hence  is  as  giver,  in  i  5)  » 

The  inequalities  of  (7 j  show  that  MinX  'Z  V.  rne  argument  above 

\ 

shows  thut  wi in  4>  ^  V.  Hence  we  have  equality* 


If,  for  example,  o  %  (*tS)T  #0, 

the  solution  is  non-unique,  to  show  tins,  let  us  observe  that  the  above 
conditicu  implies  tnat  the  values  ol'  the  linear  form  at  the  v-.rtioes  are 
such  that  one  lies  strictly  above  the  other  two.  It  loilows  that  if  we 

4 

perturb  <$>  by  a  sufficiently  sraail  amount  over  an  interior  interval  in 
such  a  way  as  to  ueave  its  total  integral  over  \G,T)  unchanged,  the  efitct 

on  L  will  be  to  leave  its  values  at  the  vertices  unchanged.  ri6nce  we  can 

% 

obtain  arbitrarily  many  solutions  from  the  original  solution  4 


it  is  also  possible  to  derive  various  conriltior.3  of  julte  special 
nature  which  will  ensure  uni  ,uen6sSo 
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Incidentally ,  the  foregoing  minimization  problem  has  a  determinate 
two-rarson  zero  sum  game  interpretation  in  which  the  minimizing  player 
picks  the  £  (subject  to  the  some  constraints  as  above),  and  the  maxi¬ 
mizing  player  picks  a  point  (u,v)  in  the  triangle  0  *.  u  *  v  ^T,  with  the 

,  *  r  i 

payoff  given  by  Ut  +  ,  The  value  of  the 

O  O 

game  is  given  by  l  5),  an  optimal  strategy  for  the  minimizing  player  is 
given  by  (6),  while  the  maximizing  player  has  an  optimal  pure  strategy 
consisting  of  choosing  the  vertex  which  yields  Uie  largest  value  for  the 
linear  form,L  . 


Alternative  Derivation  of  the  Decision  /unction. 

Let  us  in  this  section  show  now  the  Johnson  criterion,  given  In 
Theorem  2,  may  by  derived  by  use  of  the  functional  equation  approach  of 
tne  theory  of  dynamic  progra.-rm  ing. 

Let 

ll)  fia,  ,  b,  ,a^,bv, . .  .,aN,bj  ;tj  time  consumed  processing  the  N 

items  with  required  times  a  ,b 
on  the  first  and  second  machines 
when  tne  second  macnine  is  committed 
for  t  hours  ahead,  and  an  optimal 
scheduling  procedure  is  employed. 


Then  if  th6 
(2)  f  (  a,  ,  b,  ,  •  •  e 

if  we  choose  the 

13) 


first  item  is  processed  first,  we  have 
, ad  » bM  ; t j  *  a, «-  f(0,t/,a1>,bx,...,aw  ,bg;b,t  Uax(t-a,  ,oj  ). 
second  item  to  follow,  we  obtain 

c  c*3jbl3..  bA4  fta#  [  fa,- 

cj  )  . 


On  the  other  hand,  if  we  interchange  the  orders,  we  obtain 
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^4)  o-v*-  o,  t  4  C  °>°.>0J0j  «»,  bi  j  ••  •  j  bn  ^  b.  -*•  Mo*  ^  kv-  c<v  ♦ 

It  follows  from  these  formula*  that  the  order  of  the  Items  which  mini¬ 
mises  the  new  t-terc  is  optimal*  It  is  not  immediately  obvious  that  this 
order  is  independent  of  t*  W*  have 

(51  *>1  +  Ma*  {_  b,-H»  fAo*  Ct-a.jCii  J°3  *  N\<v*  I  M  o/jltt-C'.jO  5  v  b,  j  -t  b.^bv-C'^ 

=  K\  £  t  -  C*#J  0  j  4v-b,"]  ^ 

-  tAojf.  atJ  a,+vb,j  *  C4,-ea. 


It  is  easily  seen  from  this  that  regardless  of  the  value  of  t, 
the  interchange  which  minimizes  Mux  ^ c.LJ  a, ♦  -b cannot  increase 
the  total  time,  and  may  decrease  it. 


The  remainder  of  the  proof  proceeds  as  in  Johnson's  puj6r 
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